Scanning laser projection display devices and methods for projecting one or more images onto a surface with a light-scanning optical fiber

ABSTRACT

Image projection devices, high-speed fiber scanned displays and related methods for projecting an image onto a surface and interfacing with the projected image are provided. A method for projecting one or more images and obtaining feedback with an optical input-output assembly is provided. The input-output assembly comprising a light-scanning optical fiber and a sensor. The method includes generating a sequence of light in response to one or more image representations and a scan pattern of the optical fiber, articulating the optical fiber in the scan pattern, projecting the sequence of light from the articulated optical fiber, and generating a feedback signal with the sensor in response to reflections of the sequence of light.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/054,428, filed May 19, 2008, entitled “Scanning Laser ProjectionDisplay for Small Handheld Devices,” the full disclosure of which isincorporated herein by reference.

BACKGROUND

The present invention relates generally to image projection devices, andmore specifically to high-speed fiber scanned displays and relatedmethods for projecting an image onto a surface and, in some embodiments,interfacing with the projected image.

The increasing computational power and connectivity of mobile devices,such as cell phones, video iPods, and PDAs, have enabled mobile internetaccess and playback of images and video. Presently, many such electronicdevices include a built in video display, which impacts the size andcost of the device. For example, a typical hand held gaming deviceincludes a display that is sufficiently large to display enoughinformation, as well as provide an image suitable for visual enjoyment.

Projectors using LCD panels as imaging elements are reasonably low cost,but are not energy efficient. To create dark pixels in the projectedimage, LCD panels block a portion of the light energy generated by thesource from reaching the screen, so dark images require as much energyto project as bright images, placing an additional drain on the limitedbattery capacity of mobile devices. Furthermore, the maximum resolutionof the projector is constrained by the size of the LCD panel and theattainable pixel pitch. In order to increase resolution, the dimensionsof the LCD panel in the mobile device must be increased proportionately.

Holographic projectors can provide the advantage of lowered powerconsumption, as the majority of the light generated by the laser sourcesreaches the screen. However, holographic image generation requirescomplex processing, placing high computational demands on mobile deviceswith small, low-voltage processors, and the resolution of the projectedimage is also ultimately dependent on the size and pixel pitch of thespatial light modulator used to generate the holographic image.

The size and cost of mobile electronic devices is also influenced byinput features. Existing mobile devices may include a number of inputfeatures, for example, input keys, touch screens, and the like. Theinput features employed with any particular mobile device can also havea significant influence on the level of functionality provided.

While many existing mobile electronic devices provide a good bit offunctionality for their size and/or cost, further improvements in size,cost, and/or functionality are desirable. Thus, a need exists forimproved image display and improved input features, which can beemployed in various devices, for example, mobile electronic devices.

BRIEF SUMMARY

The following presents a simplified summary of some embodiments of theinvention in order to provide a basic understanding of the invention.This summary is not an extensive overview of the invention. It is notintended to identify key/critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentsome embodiments of the invention in a simplified form as a prelude tothe more detailed description that is presented later.

In many embodiments, improved methods, devices, and systems forprojecting an image are provided. In many embodiments, methods, devicesand systems for projecting an image and generating feedback via theprojected image are provided. In many embodiments, such methods,devices, and systems include an articulated fiber assembly from which asequence of light is projected to form an image, and in someembodiments, include a sensor for measuring reflections from theprojected sequence of light. The use of the disclosed articulated fiberbased projection, and projection and feedback, may provide for morecompact display and feedback functionality, which can be employed in arange of electronic devices, for example, mobile electronic devices. Inmany embodiments, the combination of projection and feedback providesfor combined input and output functionality via a compact device, whichmay be used to provide for increased functionality while simultaneouslyreducing relative size, weight, and cost.

In one aspect, a method for projecting one or more images with aprojection device comprising a light-scanning optical fiber is provided.The method includes generating a sequence of light in response to one ormore image representations and a scan pattern of the optical fiber,articulating the optical fiber in the scan pattern, and projecting thesequence of light from the articulated optical fiber. The articulatedoptical fiber is scanned within a within a Q factor of a resonantfrequency of the optical fiber.

In another aspect, a method for projecting one or more images andobtaining feedback with an optical input-output assembly is provided.The input-output assembly comprising a light-scanning optical fiber anda sensor. The method includes generating a sequence of light in responseto one or more image representations and a scan pattern of the opticalfiber, articulating the optical fiber in the scan pattern, projectingthe sequence of light from the articulated optical fiber, and generatinga feedback signal with the sensor in response to reflections of thesequence of light.

In another aspect, a system for projecting one or more images isprovided. The system includes a projection device comprising alight-scanning optical fiber, a laser assembly coupled with thelight-scanning optical fiber and operable to generate the sequence oflight, and a processor coupled with the light-scanning assembly and thelaser assembly. The projection device is operable to project a sequenceof light from the light-scanning fiber according to a scan pattern. Theprocessor comprises a tangible medium comprising instructions that whenexecuted cause the processor to generate the sequence of light inresponse to one or more image representations and the scan pattern, andprovide control signals to the laser assembly and the light-scanningassembly to project the sequence of light according to the scan pattern.

In another aspect, a system for projecting one or more images andobtaining feedback is provided. The system includes a projectionassembly comprising a light-scanning optical fiber, a sensor operable tomeasure reflections from light projected by the light-scanning opticalfiber and generate a feedback signal corresponding thereto, a laserassembly coupled with the light-scanning optical fiber and operable togenerate the sequence of light, and a processor coupled with thelight-scanning assembly, the sensor, and the laser assembly. Theprojection assembly is operable to project a sequence of light from thelight-scanning optical fiber according to a scan pattern. The processorcomprises a tangible medium comprising instructions that when executedcause the processor to determine the sequence of light in response toone or more image representations and the scan pattern, provide controlsignals to the laser assembly and the light-scanning assembly to projectthe sequence of light from the light-scanning optical fiber according tothe scan pattern, and receive a feedback signal generated by the sensorin response to reflections of the sequence of light.

For a fuller understanding of the nature and advantages of the presentinvention, reference should be made to the ensuing detailed descriptionand accompanying drawings. Other aspects, objects and advantages of theinvention will be apparent from the drawings and detailed descriptionthat follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 diagrammatically illustrates a projection and feedback system forprojecting an image onto a target area and capturing reflected lightfrom the target area, in accordance with many embodiments.

FIG. 2 diagrammatically illustrates a projection assembly that includesan articulated optical fiber for projecting an image onto a target area,in accordance with many embodiments.

FIG. 3 illustrates a drive signal for articulating an optical fiber usedto project an image and a spiral scan pattern of an articulated opticalfiber, in accordance with many embodiments.

FIG. 4 is a simplified block diagram illustrating a method forprojecting an image onto a target area while simultaneously receivingfeedback related to the projected image, in accordance with manyembodiments.

FIG. 5 is a simplified block diagram illustrating a method forprojecting an image onto a target area, in accordance with manyembodiments.

FIG. 6 is a simplified block diagram illustrating a method forcontrolling the intensity of one or more light sources in a projectionsystem, in accordance with many embodiments.

FIG. 7 is a simplified block diagram illustrating a method for writingan image to a buffer and reading the image from the buffer according toa scan pattern, in accordance with many embodiments.

FIG. 8 is a simplified block diagram illustrating a method for matchingthe frame rate of a projection system to a video frame rate, inaccordance with many embodiments.

FIG. 9 is a simplified block diagram illustrating a method for changingthe field-of-view of a projection system, in accordance with manyembodiments.

FIG. 10 diagrammatically illustrates an articulated fiber projectionsystem that can project a sequence of light output from red, green andblue lasers, in accordance with many embodiments.

FIG. 11 illustrates a functional relationship between peak drive signaland resulting throw angle for an articulated fiber projection system, inaccordance with many embodiments.

FIG. 12 illustrates the generation of the throw angle data points ofFIG. 11.

FIG. 13 illustrates a projection system incorporated into a writinginstrument, in accordance with many embodiments.

FIG. 14 illustrates a projection system incorporated into a handheldelectronic device, in accordance with many embodiments.

FIG. 15A illustrates the use of fringe areas of a projection area todisplay additional image content, in accordance with many embodiments.

FIG. 15B illustrates the dynamic orientation of a projected image inresponse to projection device roll, in accordance with many embodiments.

DETAILED DESCRIPTION

Improved methods, devices, and systems for projecting an image areprovided. In many embodiments, methods, devices and systems forprojecting an image and generating feedback via the projected image areprovided. Such methods, devices, and systems include an articulatedfiber assembly from which a sequence of light is projected to form animage, and in some embodiments, include a sensor for measuringreflections from the projected sequence of light. The ability to measurereflections from the projected sequence of light provides the ability tocorrelate specific projections and reflections, which, in manyembodiments, enables a person to interact with the projected image so asto generate feedback to an electronic device.

Using a high-speed optical scanner and a light source, an image can beprojected onto a surface. In many embodiments, the high-speed opticalscanner comprises an optical fiber that can be articulated in a scanpattern and light is projected from the articulated fiber to form animage on a target surface. Various light sources can be used. In manyembodiments, the light source for such display includes one or morelaser light sources. A laser can provide a small diameter beam that isprojected from the optical scanner so as to project an image. The smallbeam diameter can be used to generate a high resolution image. With sucha projection device, virtually any surface (e.g., opaque surface) can beused for displaying a projected image. By varying the distance betweenthe projection device and a target surface, the size of the displayedimage can be varied. For example, greater distances will increase thedisplayed image size, whereas smaller distances will decrease thedisplayed image size. In many embodiments, light source intensity, imagesize, scanning pattern, and/or scanning parameters are varied and/ormodified to achieve desired image characteristics. In many embodiments,the projected image is configured to correctly display on an obliquetarget surface and/or within a targeted display area. Such variationsand/or modifications can be used to achieve desired imagecharacteristics for a variety of target surface characteristics (e.g.,size, orientation, reflectivity) and ambient conditions (e.g.,illumination).

Various types of optical scanning systems can be utilized in order toprovide the desired performance and image generation. Exemplary systemsinclude those having MEMS two axis mirror scanners, as well as opticalfiber scanners.

In addition to providing a display, the laser and optical scanner canprovide an input function. In use, as the optical scanner scans the beamof light across the display surface, a certain amount of light isreflected back, for example, into the optics of the display generator.This reflected light can be detected and measured. By correlating wherethe beam is in the displayed image, for example, and the amount ofreflected light, certain decisions can be performed. Thus, an image canbe displayed and certain interactions (e.g., physical interactions) withthe image can be detected. For example, if the image of a keyboard wereto be displayed on a surface, and then a finger is placed on a displayedkey, the amount of reflected light would change or be detectable, andthis would indicate a touched key. Numerous uses and examples areavailable. Another example would be for a gaming device. Instead ofusing buttons on the device to interact with the device, the user coulduse their fingers, hand, body, or other objects for input function.

In addition to the light projected to form a visible image, one or morenon-visible wavelengths can also be projected, for example, continuouslyso as to “illuminate” the target area. Reflections from a non-visiblewavelength can be used to track the movement of objects (e.g., a fingeras discussed above) that are interacting with the projected image usingknown methods (e.g., known machine vision techniques). As such, the useof a non-visible wavelength can provide for tracking of an objectinteracting with the projected image in a way that is less dependentupon the particular image being projected.

Using display and input techniques described herein, very small devicesare possible. One such possible device (e.g., gaming device) includes aprojection device (e.g., an FSD) built into a shape similar to an inkpen (either a functioning ink pen or a non-functioning ink pen). Otherpossible devices are handheld electronic devices, such as cell phones,PDAs, laptops, palmtop computers, watches, GPS hand units, and the like.

A projection device (e.g., an FSD) as described herein can have manyuses. As further explained, in many embodiments the projection devicehas substantially simultaneous input and output functions. Since it hassubstantially simultaneous input and output functions, the projectiondevice can be used for an array of uses. For example, it could be usedto read bar codes and then display information about the bar coded item.In another example, it can be used for scanning a printed text or imageon a page to be output later by projection, or to another device such asa computer. Such a device configured for reading text and outputtingusing a built in speaker may be particularly useful, for example, forthose that are visually impaired.

FIG. 1 schematically illustrates a projection system 10 for projectingone or more images on a target area and detecting and/or measuringreflected light from the target area, in accordance with manyembodiments. The projection system 10 may be used in a wide range ofapplications, for example, mobile phones, laptops, personal digitalassistants (PDAs), MP3 players, smart phones, digital cameras,camcorders, personal mobile televisions, portable projection units, GPSenabled devices, and automobiles. The projection system 10 includes afunction processor with memory 12, output/input electronics 14, anoptical scanner driver 16, an optical scanner 18, projection/receiveoptics 20, output/input optics 22, a sensor 24, a light source 26, alight source driver 28, and an audio output/input 30. In manyembodiments, one or more of these elements are omitted. For example, inmany embodiments, the audio output/input 30 is omitted. In manyembodiments, the sensor 24 is omitted, etc.

The function processor 12 includes one or more microprocessors and/orone or more dedicated electronics circuits, which may include a gatearray (not shown). The function processor 12 can also include scannerdrive electronics, sensor amplifiers and A/D converters (not shown). Inmany embodiments, the function processor 12 controls actuation of theoptical scanner 18 by sending electrical control signals to the opticalscanner driver 16. In many embodiments, the function processor 12 alsocontrols activation of the light source 26 by sending electrical controlsignals to the light source driver 28. Simultaneous actuation of theoptical scanner 18 and activation of the light source 26 results in animage 32 being projected for display on a target area. As will beappreciated by those of skill in the art, the methods and techniques ofthe present invention may be carried out by software modules executed bythe function processor 12 and/or by electronic hardware provided in thefunction processor 12.

In many embodiments, the function processor 12 is in communication withother elements of the projection system 10 via output/input electronics14. The output/input electronics 14 allows for electrical communicationbetween the function processor 12 and the other elements of theprojection system 10 by providing appropriate interfaces as known in theart.

The optical scanner 18 functions to project an image 32 onto a targetarea via the projection/receive optics 20. In many embodiments, theprojection/receive optics 20 include a lens assembly for directing andfocusing light directed out of the optical scanner 18 onto the targetarea. The projection/receive optics 20 may also direct and focus lightreflected from the target area to the optical scanner 18 and/or to thesensor 24. Accordingly, the lens assembly in the projection/receiveoptics 20 may also direct and focus light reflected from the target areaonto the optical scanner 18 and/or onto the sensor 24.

The optical scanner driver 16 communicates electrical drive signals tothe optical scanner 18 for actuating the optical scanner 18 according toan electrical control signal received from the function processor 12. Inmany embodiments, the drive signals include two drive signals foractuating a cantilevered optical fiber in a scan pattern about twoseparate axes.

The light source 26 can be any light source used for projecting theimage 32. For example, in many embodiments, the light source 26 is oneor more lasers, for example, a red laser, a green laser, and a bluelaser, the output of which can be combined as required to producevarious image colors. The light source 26 can emit a continuous streamof light, modulated light, or a stream of light pulses. The light source26 can comprise a plurality of different light sources. For example, theplurality of different light sources can include one or more of a redlight source, blue light source, green light source (collectivelyreferred to herein as an “RGB light source”), an infrared (IR) lightsource, an ultraviolet (UV) light source, and/or a high-intensity lasersource. The light sources can be configured to be switchable between afirst mode (e.g., continuous stream) and a second mode (e.g., stream oflight pulses). If a plurality of light sources are used, a combiner canbe used as described below.

In many embodiments, the light source driver 28 communicates electricalcontrol signals to the light source 26 for activating the one or morelight sources constituting the light source 26 according to anelectrical control signal received from the function processor 12.

The projection system 10 can optionally include output/input optics 22for communicating light from the light source 26 to the optical scanner18 and/or for communicating reflected light to the sensor 24. Theoutput/input optics 22 can be a lens assembly, waveguide, or other meansfor communicating light as is known in the art.

The projection system 10 can optionally include a sensor 24, for examplea photo sensor or other known sensor, for detecting light reflected fromthe target area. The sensor 24 detects, and optionally measures, lightreflected from the target area. The sensor 24 can include one or morephoto diodes and the like as is known in the art. In many embodiments,the sensor 24 converts the reflected light into an electrical feedbacksignal and communicates the electrical feedback signal to the functionprocessor 12 via the output/input electronics 14.

In many embodiments, the sensor 24 does not receive light communicatedvia the output/input optics 22. Rather, the sensor 24 may be placedanywhere where it can detect, and optionally measure, light reflectedfrom the image projected on the target area, for example, closer to theimage 32. For example, in many embodiments, the sensor 24 is locatedadjacent the optical scanner 18 so as to receive reflected lightcommunicated only through the projection/receive optics 20. In manyembodiments, the sensor 24 is located adjacent the optical scanner 18 soas to directly receive reflected light.

In many embodiments, the projection system 10 includes an audiooutput/input 30. The audio output/input 30 can include a speaker, anearphone, a microphone, and the like.

The function processor 12 can be coupled together with a memory. In manyembodiments, the memory is separate from the function processor 12. Thememory can be used for storing software modules, look-up tables, andalgorithms that control the operation and any calibration of theprojection system 10. In many embodiments, a control routine is used bythe function processor 12 to control the optical scanner 18 and lightsource 26. The control routine can be configurable so as to matchoperating parameters of the optical scanner 18 (e.g., resonantfrequency, voltage limits, zoom capability, color capability, etc.). Thememory can also be used for storing images to be displayed by theprojection system 10. As noted below, the memory may also be used forstoring data concerning reflections from the target area, thresholdvalues for each pixel of the stored images, parameters of the opticalscanner 18, etc.

For ease of reference, other conventional elements that can be includedin the projection system 10 are not shown. For example, many embodimentsof the projection system 10 of the present invention will typicallyinclude conventional elements such as amplifiers, D/A converters and A/Dconverters, clocks, waveform generators, and the like.

In many embodiments, a projection device can include technologydeveloped for a scanning fiber endoscope (SFE) as described in numerouscommonly owned U.S. patent applications and patents, which, for example,include: U.S. Pat. No. 7,298,938, entitled “Configuration Memory for aScanning Beam Device”, filed on Oct. 1, 2004; U.S. patent applicationSer. No. 10/956,241, entitled “Remapping Methods to Reduce Distortionsin Images,” filed on Oct. 1, 2004; U.S. Pat. No. 7,159,782, entitled“Methods of Driving a Scanning Beam Device to Achieve High Frame Rates,”filed on Dec. 23, 2004; U.S. patent application Ser. No. 11/969,141,entitled “Methods of Driving a Scanning Beam Device to Achieve HighFrame Rates,” filed on Jan. 3, 2008; U.S. Pat. No. 7,189,961, entitled“Scanning Beam Device with Detector Assembly,” filed on Feb. 23, 2005;U.S. patent application Ser. No. 11/094,017, entitled “Methods andSystems for Creating Sequential Color Images,” filed on Mar. 29, 2005;U.S. Pat. No. 7,312,879, entitled “Distance Determination in a ScannedBeam Image Capture Device,” filed on Aug. 23, 2005; U.S. Pat. No.7,395,967, entitled “Methods and Systems for Counterbalancing a ScanningBeam Device,” filed on Jul. 21, 2005; and U.S. patent application Ser.No. 12/040,249, entitled “Piezoelectric Substrate Fabrication andRelated Methods,” filed on Feb. 29, 2008; the complete disclosures ofwhich are incorporated herein by reference. In many embodiments, a SFE,e.g., as described in the referenced publications, can be modified foruse as projection device or a display, which can be referred to as aFiber Scanned Display (FSD). The FSD has been demonstrated using an SFEprobe in which the laser light is modulated such that an image can bedisplayed on a surface. It uses many of the techniques, such asremapping, that are used in the SFE. There are a number ofimplementation issues that are specific to the FSD. These issues and theunique inventive aspects to address these issues are described herein.

One issue is that the fiber and hence the projected spot may move atdifferent speeds during the specific scan pattern used. For example,when a spiral scan pattern is employed, the projected spot may moveslower in the center of the spiral scan and increase in speed as thescan moves outward. In some instances, this can create an image that isbright in the center and darker as it moves outward. In manyembodiments, the brightness of the projected light is increased as thescan moves outward thereby providing an improved image for display. Theincrease can be selected to be proportional to the speed of the spot.This can include changing display timing or signal multiplexing.

It should be appreciated that a wide variety of scan patterns can beemployed, for example, a spiral scan pattern, an expanding ellipsepattern, or a Lissajous pattern. The particular scan pattern used can beselected to provide desired image characteristics, for example,resolution, time-dependent surface coverage, intensity, etc.

Another issue addressed is that video is normally generated in a rasterpattern which does not fit many of the scan patterns that can be used inan FSD. In many embodiments, an incoming frame of video is stored in amemory buffer and read out of the buffer in the desired manner, forexample, according to the scan pattern used. This can includeintroducing a time delay, such as a one frame time delay (though, couldbe slightly less) but will not be a problem for typical video. The videobuffer can be double buffered. In many embodiments, for example, aprojection system includes a buffering means that includes two pieces ofmemory, where the system would write into a first memory, and outputfrom a second memory.

Another issue addressed is that the frame rate may not match the scannerframe rate as each scanner has a different resonant frequency. In manyembodiments, a projection device/FSD can be configured to adjust thenumber of settling cycles dynamically to match the video frame rate.

FIG. 2 illustrates a projection assembly that can be used in aprojection system, in accordance with many embodiments. In thisprojection assembly, the optical scanner 18 includes an optical fibermounted within a housing 34. As shown, the optical fiber comprises aproximal portion 36 and a cantilevered distal portion 38 that comprisesa distal tip 40. The optical fiber is fixed along at least one point ofthe optical fiber so as to be cantilevered such that the distal portion38 is free to be deflected. In many embodiments, the optical scanner 18includes one or more cantilevered optical fibers. In many embodiments,the optical scanner 18 includes other means by which to project light ina scan pattern, for example, deflectable mirrors, amicro-electromechanical system (MEMS), a galvanometer, a polygon,multiple optical elements moved relative to each other, or the like.While the remaining discussion focuses on scanning fiber devices thatare used for displaying images on a target site and, in manyembodiments, receiving feedback concerning the generated images, it willbe appreciated that alternate scanning means, for example, theaforementioned devices, can be used instead of a scanning fiber device.

The cantilevered distal portion 38 can have any desired dimensions andcross-sectional profile. For example, the distal portion 38 can have asymmetrical cross-sectional profile or an asymmetrical cross-sectionalprofile, depending on the desired characteristics of the projectionassembly. A distal portion 38 with a round cross-sectional profile willtypically have substantially the same resonance characteristics aboutany two orthogonal axes, while a distal portion 38 with an asymmetriccross-sectional profile (e.g., an ellipse) will typically have differentresonant frequencies about the major and minor axes. If desired, thedistal portion 38 can be linearly or non-linearly tapered along itslength.

To articulate the distal portion 38, the distal portion 38 is coupled tothe piezo drive electronics 14, which supply one or more drive signalsto a fiber actuator 41. The drive signals supplied by the opticalscanner driver 16 cause the fiber actuator 41 to actuate the distalportion 38 in a scan pattern. The scan pattern can be one dimensional orconsist of a plurality of dimensions. In many embodiments, the scanpattern is two dimensional. In many embodiments, the distal portion 38is driven at a frequency that is within a Q-factor of the resonantfrequency of the distal portion 38, and preferably at its mechanical orvibratory resonant frequency (or harmonics of the resonant frequency).As can be appreciated, the distal portion 38 does not have to be drivenat substantially the resonant frequency. However, if the distal portion38 is not driven at its resonant frequency, a larger amount of energy isrequired to provide a desired radial displacement of the distal portion38. In many embodiments, the fiber actuator 41 is a patternedpiezoelectric tube. An electrical drive signal from the optical scannerdriver 16 causes the piezoelectric tube to actuate the distal portion 38in a scan pattern so that light emitted from the distal tip 40 isprojected to the target area in a desired pattern. The fiber actuator 41need not be a patterned piezoelectric tube. Rather, in many embodiments,the fiber actuator 41 includes a permanent magnet, an electromagnet, anelectrostatic drive, a sonic drive, an electro-mechanical drive, or thelike.

In many embodiments, the housing 34 surrounds the distal portion 38 andthe fiber actuator 41. The fiber actuator 41 can be mounted within thehousing 34 via one or more collars 42. The housing 34 can also house allor a portion of the projection/receive optics 20. The projection/receiveoptics can be spaced from the distal end 40 of the distal portion 38 soas to focus light emitted from the distal end 40 to, for example,provide better resolution and/or provide an improved field of view forthe projection assembly. One or more of the lenses of theprojection/receive optics 20 can be fixed relative to the distal end 40of the distal portion 38 and/or one or more of the lenses of theprojection/receive optics 20 can be movable relative to the housing 34.

Although not illustrated in FIG. 2, in many embodiments the sensor 24(shown in FIG. 1) is disposed within the housing 34. Exemplarytechniques for incorporating a sensor 24 are described in U.S. Pat. No.7,189,961, the complete disclosure of which was incorporated herein byreference above. In many embodiments, the sensor 24 is located so as toreceive reflected light via the optical fiber. In such embodiments, theoptical fiber functions to receive light reflected from the target area.A technique for using the optical fiber to receive light reflected fromthe target area is described in U.S. Pat. No. 7,189,961. In manyembodiments, the sensor 24 is located outside the optical path of theoptical fiber. For example, the sensor 24 can be located adjacent theoptical fiber where it may receive light reflected from the target area.In many embodiments, the sensor 24 is omitted.

In many embodiments, the optical scanner driver 16 includes piezo driveelectronics. The piezo drive electronics can include one or moremicroprocessors and/or one or more dedicated electronics circuits whichmay include a gate array for driving piezoelectric tubes. The opticalscanner driver 16 can be provided inside or outside of the housing 34.When the optical scanner driver 16 is provided outside of the housing34, wires 44 can be used to couple the optical scanner driver 16 withthe fiber actuator 41. Providing the optical scanner driver 16 outsideof the housing 34 advantageously increases the configurability of theprojection system 10 thus increasing the ability to implement theprojection system 10 in various applications.

In many embodiments, the light source 26 includes one or more laserdiodes. For example, the light source 26 can be one or more lightemitting diodes, which can be used to communicate light through fiberoptic cables as is known in the art. The light source 26 can be locatedinside or outside of the housing 34. When the light source 26 isprovided outside of the housing 34, an optical path 46 (e.g., an opticalfiber, or optical cable) can be used to couple the light source 26 withthe distal portion 38. Locating the light source 26 outside of thehousing 34 advantageously increases the configurability of theprojection system 10 thus increasing the ability to implement theprojection system 10 in various applications.

FIG. 3 illustrates a two dimensional “spiral” scan pattern 48 that canbe used in accordance with many embodiments. In many embodiments, thedistal portion 38 is actuated to project light in the spiral scanpattern 48 through the use of a horizontal sinusoidal vibration drivesignal and a vertical sinusoidal vibration drive signal, both of whichhave time domain voltage patterns similar to the sinusoidal drive signal50. The functional processor 12 can be used to control the opticalscanner driver 16 such that the optical scanner driver 16 applies thesedrive signals to the fiber actuator 41. In many embodiments, thesinusoidal drive signal 50 is amplitude modulated in a triangle patternas shown. In many embodiments, a horizontal and a vertical drive signalsare driven with a 90 degree phase shift between them. In manyembodiments, such drive signals cause a sequence of light to beprojected in a pattern that starts at a central point and spiralsoutward until a maximum diameter circle is created. The maximum diameterof the circle is a function of the peak amplitude of the sinusoid at thetop of the ramp (and the mechanical properties of the cantileveredportion 38).

The sinusoidal drive signal 50 comprises three portions: an imagingportion 52, a retrace portion 54, and a settling portion 56. During theimaging portion 52, the projected sequence of light spirals outward. Theimaging portion 52 is typically used for projecting an image 32. Duringthe retrace portion 54, the distal portion 38 spirals inward. In manyembodiments, an image is not projected during the retrace portion 54.However, in many embodiments, the retrace portion 54 is used to projectan image 32. During the settling portion 56, the distal portion 38 issubstantially at rest in the center of the spiral pattern. Therefore, inmany embodiments, no image is projected during the settling portion 56.However, the settling portion 56 can optionally be used to project anillumination spot in the center of the spiral pattern.

As can be appreciated, back scattered light (e.g., light reflected froma target area) can be detected at any time projected light is reflected.In other words, back scattered light can be detected during the imagingportion 52, the retrace portion 54, and/or the settling portion 56. Ascan be further appreciated, the spiral scan pattern is merely oneexample of a scan pattern and other scan patterns, such as an expandingellipse pattern, a Lissajous pattern, a rotating propeller scan pattern,a raster scan pattern, a line pattern, and the like, can be used toproject a sequence of light onto a target area so as to form an image.

FIG. 4 is a simplified block diagram illustrating a method forprojecting an image on a target area while simultaneously receivingfeedback related to the projected images. In use, as the optical scanner18 projects a sequence of light in a scan pattern towards a target areaon a display surface, a certain amount of light is reflected from thetarget area. This reflected light can be detected and measured. Bycorrelating where the projected light is in the scan pattern with thereflected light detected/measured, information relating to the image andany interactions with the image can be determined. For example, an imagecan be displayed and certain interactions (e.g., physical interactions)with the image can be detected. For example, if the image of a keyboardwere to be displayed on a surface, and then a finger is placed on adisplayed key, the amount of reflected light would change or bedetectable, which can be used to indicate a touched key. A wide varietyof uses are possible. As a further example, such an approach can be usedto provide a user the ability to interact with a gaming device byinteracting with a projected image using their fingers, hand, body, orother objects instead of physical input features on the gaming device.

In step 60, an image is read from memory. For example, the functionprocessor 12 can be used to read an image from memory. The image can bea single image, or one of a series of images within a sequence of imagesforming a video.

In step 62, a sequence of light is generated in response to the imageand a scan pattern. For example, in many embodiments, the functionprocessor 12 can process the image read from memory on a pixel-by-pixelbasis. Each of the pixels in the image read from memory can be processedto generate a resulting sequence of light to be projected via the scanpattern. Depending upon the resolution of the image read from memory andthe resulting resolution of the image projected, any particular portionof the generated sequence of light can be a function of one or more ofthe pixels of the image read from memory.

In step 64, the sequence of light is projected from an optical scanneraccording to the scan pattern. For example, in many embodiments, thefunction processor 12 controls the light source 26 via the light sourcedriver 28 to output the sequence of light to the optical scanner incoordination with the scan pattern of the optical scanner. In manyembodiments, the light sequence is output from the light source 26synchronously with the drive signal used to actuate the optical scanner.In many embodiments, the light source 26 emits a single color of lightso as to project a monochrome image. In many embodiments, the lightsource 26 emits various combinations of red, green, and blue light so asto output a color image. As a result, an image can be formed on a targetarea to which the optical scanner 18 is directed.

In step 66, a sensor is used to generate a feedback signal in responseto reflections of the projected sequence of light. For example, for oneor more locations of the image formed, reflections from the target areacan be detected at practically the same time as the portion of the lightsequence projected onto the location is projected. In many embodiments,the reflections can be processed to determine wavelengths and/orintensities of light reflected from the one or more locations.Information concerning the detected/measured reflections, such aswavelength and/or intensity information, can be stored in memory coupledto or separate from the function processor 12.

In step 68, the feedback signal is processed. For example, for one ormore locations of the image formed, the function processor 12 canprocess the feedback signal to determine whether a user interactionexists. In many embodiments, the function processor 12 determineswhether a user or object has interacted image by processing both theportion of the projected sequence of light corresponding to thereflection and the measured reflection. In many embodiments, anon-visible wavelength of light can be continuously projected so as to“illuminate” the target area so that reflections of the non-visiblewavelength can be used to track user interactions relative to the image,for example, using known machine vision approaches. Such detected userinteractions with the projected image can be used for a wide variety ofpurposes as will be appreciated by one of skill in the art, and a widevariety of actions can be taken upon detecting a user interaction,including, for example, user notification.

In many embodiments, the feedback signal can be processed so as to avoidnuisance indications of user interaction with the projected image. Forexample, if a user interaction is indicated via the reflections measuredfor only a single location, this may be seen as an insufficient basis onwhich to determine the existence of a user interaction with theprojected image. Instead, a user interaction determination can be basedon a plurality of reflections from a corresponding plurality oflocations. As can be further appreciated, a user need not be notified atall when a threshold value is exceeded. Rather, the function processor12 may internally store such information, set flags, etc., to be usedfor other processing decisions.

Numerous techniques exist for the function processor 12 to determine ifa user interaction exists. Although certain embodiments follow, otherapproaches are possible as will be recognized by one of skill in theart.

In many embodiments, the function processor 12 determines an expectedreflection characteristic of the corresponding portion of the projectedsequence of light and compares the expected reflection characteristicwith the measured reflected light characteristic. In many embodiments,the characteristic includes one or more of intensity, color, etc. If adifference between the expected reflection characteristic and themeasured reflection characteristic does not exceed a predeterminedthreshold, then the function processor 12 determines that no userinteraction exist. On the other hand, if the difference between theexpected reflection characteristic and the measured reflectioncharacteristic exceeds the predetermined threshold, then the functionprocessor 12 determines that user interaction exists.

In many embodiments, the function processor 12 determines thecharacteristic of a plurality of actual reflected images for aparticular location. The characteristics can be averaged over time. Thefunction processor 12 then compares a current measured characteristic ofthe reflection for the particular location with a previous or theaverage measured characteristic of the reflection for the particularlocation. If a difference between the currently measured characteristicand the previous or average measured characteristic does not exceed apredetermined threshold, then the function processor 12 determines thatno user interaction exists. On the other hand, if the difference betweenthe current measured characteristic and the previous or average measuredcharacteristic does exceed the predetermined threshold, then thefunction processor 12 determines that a user interaction exists. Thisapproach may be used in addition to or alternatively to the approachdescribed above using expected reflection characteristics.

In many embodiments, a non-visible wavelength is projected so as to“illuminate” the target area. Reflections of the non-visible wavelengthcan be used to track movements within the targeted area, which can beused to indicate user interactions. Known machine vision approaches canbe used to track movements within the targeted area using knownapproaches. Such non-visible wavelengths can include a variety ofwavelengths, for example, one or more infrared wavelengths.

FIG. 5 illustrates a method of projecting images that can be used inaccordance with many embodiments. In step 70, an image is read frommemory. For example, in many embodiments, the function processor 12reads an image from the memory. The image can be a still-image or asingle image within a sequence of images for forming a video.

In step 72, a sequence of light is generated in response to the imageand a scan pattern. In many embodiments, step 72 can be accomplishedsimilar to step 62 discussed above.

In step 74, the sequence of light is projected from an optical scanneraccording to the scan pattern. In many embodiments, step 74 can beaccomplished similar to step 64 discussed above.

FIG. 6 illustrates an improvement to step 74, which can be used inaccordance with many embodiments. One characteristic of images outputfrom an optical scanner employing a spiral scan pattern is that theimage may be brighter in the center of the image and darker towards theouter edges of the image. This characteristic is due to the opticalscanner moving at a lower velocity when located closer to the center ofthe spiral scan compared to a velocity when located further away fromthe center of the spiral scan. Modifications to the method illustratedin FIG. 5 can improve the quality of the image output from the opticalscanner by selectively adjusting the intensity of the sequence of lightprojected.

In step 80, the location of the scanning element is determined. In manyembodiments, as the optical scanner is actuated in patterns such as aspiral pattern, the function processor 12 can determine the location ofthe optical scanner within the scan pattern. The location of the opticalscanner can be determined in a variety of ways. For example, in manyembodiments, the function processor 12 determines the location based onthe characteristics of the optical scanner and drive signal generated bythe optical scanner driver. In many embodiments, a sensor is used tomeasure the location of the optical scanner. Regardless of the techniqueused, information relating to the location of the optical scanner can bestored in the memory coupled to or separate from the function processor12.

In step 82, the image location to be displayed is determined. In manyembodiments, the image stored in the memory comprises a plurality ofpixels, where each location to be displayed can be a function of one ormore of these pixels. The number of pixels involved can depend on therelative resolution of the image stored in memory and the resolution ofthe location of the image to be formed on account of the scan patternused.

In step 84, the amount of light to be projected for the image locationto be displayed is determined. The amount of light to be projected is afunction of the image location to be displayed and the location of theimage location within the scan pattern. The location of the imagelocation within the scan pattern impacts the rate at which the light isscanned and, for at least some scan patterns, can impact the relativeseparation between scanned locations, both of which can impact theamount of light distributed per area at the image location in question.In many embodiments employing a spiral scan pattern, the amount of thelight to be projected is increased proportionally to the distance of thelocation of the optical scanner from a position of rest for the opticalscanner. In other words, the amount of light to be projected increasesas a distance between a current location of the optical scanner and acenter location of the optical scanner increases. In many embodiments,the amount of light to be projected is increased proportional to avelocity of the optical scanner. In other words, the amount of light tobe projected increases as a velocity of the optical scanner increases.

The amount of light can be varied by varying the intensity of the lightproduced by the light source 26 and can be varied by varying the pulsewidth of the light projected. A combination of an intensity variationand a pulse width variation can also be used. In many embodimentsinvolving a single color light source, the function processor 12determines the intensity of the light source based on an intensity valueof the image location to be displayed. In many embodiments involvingmultiple light sources that include different color sources, thefunction processor 12 determines the intensity for each of the lightsources. For example, in many embodiments RGB light sources are used. Inthis case, the function processor 12 determines the intensity of the RGBlight sources based on RGB intensity values of the image location to bedisplayed. Similar approaches can be used when pulse width variation isused, or when a combination of intensity variation and pulse widthvariation is used.

In step 86, the light source is controlled to output the determinedamount of light. Various known methods for varying the amount of lightthrough intensity variation, pulse width variation, or a combination ofthe two can be used.

FIG. 7 illustrates an improvement to step 70 in the method of FIG. 5.Video is often generated in a raster pattern. Where the optical scanneris controlled to generate a non-raster scan pattern, for example, aspiral scan pattern, the scan pattern does not match the raster pattern.In many embodiments, a modifications to the method illustrated in FIG. 5involves copying an image to a buffer and reading the buffered image inaccordance with the scan pattern used.

In step 90, the image is copied from memory to a buffer. For example, inmany embodiments the function processor 12 copies sequential images froma video stored in the memory to a buffer. The buffer may be coupled toor separate from the function processor 12.

In step 92, the image is read from the buffer according to the scanpattern employed. For example, in many embodiments, the functionprocessor 12 reads the images out of the buffer in a manner appropriatefor the scan pattern, for example, a spiral scan pattern. This caninclude introducing a time delay, such as a one frame time delay(though, the delay can be slightly less). However, such a time delay isnot likely to have a detrimental affect on most video. In manyembodiments, the buffer is double buffered. In many embodiments, theprojection system 10 includes two pieces of memory (either coupled to orseparate from the function processor 12), where the projection system 10would write into a first of the two memories, and output from a secondof the two memories.

FIG. 8 illustrates in improvement to step 74 in the method illustratedin FIG. 5. The optical scanner has a frame rate that is dependent uponthe characteristics of the optical scanner, such as the resonantfrequency of the optical scanner, as well as upon the scan patternemployed. Where the projection system 10 projects a video, the video hasa frame rate based on the coding of the video. The frame rate of theoptical scanner may not be equal to the frame rate of the video. Thesteps illustrated in FIG. 8 can be used to match the frame rate of theoptical scanner with the video frame rate.

In step 100, the video frame rate is determined. In many embodiments,the function processor 12 determines the frame rate of a video to beprojected. For example, the frame rate can be determined by analyzingthe properties of the video. Information concerning the frame rate ofthe video can be stored in the memory coupled to or separate from thefunction processor.

In step 102, the scanner frame rate is determined. For example, in manyembodiments, the frame rate of the optical scanner is pre-stored in thememory. In many embodiments, the function processor calculates the framerate of the optical scanner based on user-inputted characteristics ofthe optical scanner. Information concerning the frame rate of theoptical scanner can be stored in the memory.

In step 104, the number of settling cycles required to match the videoframe rate are determined. In many embodiments, the function processordetermines the number of setting cycles necessary to match the framerate of the optical scanner to the frame rate of the video. For example,in many embodiments, the function processor determines the length of thesettling portion 56 of the sinusoidal drive signal 50 illustrated inFIG. 3. The number of settling cycles can be increased to effectivelydecrease the frame rate of the optical scanner. Accordingly, the numberof settling cycles can be varied so that the frame rate of the opticalscanner is equal to the frame rate of the video.

In step 106, the scanner is driven according to the scan pattern and thenumber of settling cycles. In many embodiments, the function processor12 controls the optical scanner driver 16 to drive the optical scanner18 according to the scan pattern and the number of settling cycles.

FIG. 9 illustrates in improvement to the method illustrated in FIG. 5.The projection system 10 projects an image having a size thatcorresponds to a maximum voltage of the electrical control signal usedto drive the optical scanner 18. For example, as illustrated in FIG. 3,the outer diameter of the spiral scan pattern 48—and thus a size of aprojected image for a given distance—is based in part on the maximumvoltage of the sinusoidal drive signal 50. Thus, variation in the drivesignal can be used to adjust the scan angle. Such a variation can beemployed in a variety of ways, for example, as desired by the user ordynamically on a frame-by-frame basis. This technique may be used tocreate a desired image size at varying projection distances. Forinstance, a user browsing the internet at a desk placed against a wallmay use a wider scan angle to produce a large image at a relativelyshort throw distance, and a narrower scan angle to project apresentation on a distant wall at a meeting. In many embodiments, throwangles ranging from 30 degrees to 100 degrees can be used.

In step 110, a first drive signal is applied to the optical scannerthereby resulting in a first throw angle. In step 112, a second drivesignal can be determined that will change the throw angle. In step 114,the second drive signal is applied to the optical scanner therebyresulting in a change in throw angle from the first throw angle. Forexample, during a single frame employing a spiral scan pattern, the rateof increase of the scan angle can be increased as the scan pattern movesfrom the center portion of the scan pattern and outward.

In many embodiments, a resolution of the projected image can be tradedoff with the frame rate of the projected image. In other words,resolution of the projected image can be increased at the cost of adecreased frame rate. Similarly, the frame rate of the projected imagecan be increased at the cost of a decreased resolution of the projectedimage. In many embodiments, the same optical scanner is used to generate240-line images at 60 Hz, 600-line images at 30 Hz, and 1000-line imagesat 15 Hz. The tradeoff occurs because the more dense scan patternsrequired to produce an increased resolution image take longer tocomplete as compared to the less dense scan patterns used to producelower resolution images. For example, with a spiral scan pattern, morespirals are needed for a higher resolution image, and the additionalspirals take additional time to complete. Advantageously, higherresolutions can be used when projecting still images whereas higherframe rates can be used when projecting video.

FIG. 10 illustrates a projection assembly in accordance with manyembodiments. The components of the projecting assembly are identical tothose illustrated in FIG. 2 except for as follows. In this projectionassembly, the housing 34 has a length (l) equal to approximately 9 mmand a diameter (d) equal to approximately 1 mm. The optical path 46 is asingle mode optical fiber, and the optical scanner driver creates andapplies sinusoidal drive signals having different maximum amplitudes atdifferent times so as to vary the throw angle as discussed above. Thelight source 26 consists of a red laser, a green laser, and a bluelaser. The output/input optics 22 includes an optical combiner thatfunctions to combine an output from the red laser, green laser, and bluelaser and communicate the combined signal to the optical path 46.

FIG. 11 illustrates the relationship between peak drive voltage andprojector throw angle for an example optical scanner comprising ascanning cantilevered fiber. The optical scanner used comprises asingle-mode optical fiber, the tip of which has been inserted into a 0.4mm diameter hollow piezoelectric actuator, as illustrated in FIG. 10.The piezoelectric actuator was vibrated at a constant 11,532 Hz scanrate, producing 500 line spirals with a 30 Hz frame rate. The peakvoltage was electronically adjusted and drive voltages that produced 20degree, 40 degree, 60 degree, 80 degree, 90 degree, and 100 degree throwangles were measured. As can be seen from FIG. 11, the relationshipbetween throw angle and peak drive voltage for the example opticalscanner is nearly linear.

FIG. 12 contains photographs of each projected spiral for the data ofFIG. 11. It is interesting to note that at very wide throw angles, thelight is passing through the extreme periphery of the projection lenses,and lateral chromatic aberration becomes apparent as shorter wavelengths(blue) are refracted at sharper angles than longer wavelengths (red).This can be corrected either optically or digitally, by a relativescaling of each color channel. One can also see shadows at the bottomedge of the 100° throw angle spiral caused by bits of adhesive at theedges of the miniature projection lenses.

The ability to vary throw angle can be used to provide a number ofsignificant benefits. For example, it may often be the case that theoptimal projected image size will be a function primarily of the ambientlighting conditions and the reflectivity of the surface. For example,for a particular lighting condition, a projected image the size of an8.5″×11″ piece of paper may be the largest projected image size thatpossesses enough luminance to be usable to the viewer, so that if theviewer moves the projector farther away from the screen (which wouldnormally increase the size of the projected image), it may be useful toautomatically reduce the throw angle to keep the image scaled to8.5″×11″ or less. This may be a fairly common scenario, for example, ifthe projector were incorporated into a laptop computer that isprojecting an image onto a wall behind a desk and the user periodicallyrepositions the laptop on the desk for ergonomic typing comfort, etc.The throw angle can also be adjusted, for example, by changing the“mode” of the projection device. For example, when a projection deviceprojects a web browsing window on the surface of a desk from a shortdistance, the throw angle can be increased to create a large usableimage from a short throw. When the device is placed in “movie mode” toproject a video on a more distant wall, the throw angle can be reducedto maintain a desired projection size with a usable image brightness.Such a mode change can be accomplished manually via user input, orautomatically. For example, the device can be configured such that whenthe appropriate digital content loads, it automatically switches modes.For example, if the user is browsing the web in a small projected windowwith a 3 by 4 aspect ratio, and the new web page contains an embeddedmovie clip, then the device can automatically switch the aspect ratio ofthe projection to match that of the clip (e.g., 16×9) and/or rescale thetotal size of the projected window.

FIG. 13 illustrates a projection system incorporated together with awriting instrument. The writing instrument can be a pen, pencil, orelectronic writing tool. The writing instrument can include a writingend 120, a body 122 for encasing the elements of the projection systemand additional elements of the writing instrument, such as a clip 124for removably attaching the writing instrument to objects, a userinterface 126 for receiving user inputs so as to allow a user to controlthe projection system, and a projection end 128 for projecting an image32. In many embodiments, the elements of the projection system and theprojection assemblies illustrated in FIG. 1, FIG. 2, and FIG. 10 can beencased in the body 122.

FIG. 14 illustrates a projection system in accordance with manyembodiments. The elements of this projection system are incorporatedtogether with a handheld electronic device, such as a mobile phone,personal data assistant (PDA), and the like. The handheld electronicdevice includes a body 130 for encasing the elements of the projectionsystem and elements of the electronic device. The housing 34 of theprojection system can be provided at a distal end 132 of the handheldelectronic device for projecting the image 32. In many embodiments, theprojection system includes a red laser 134, a green laser 136, and ablue laser 138. In many embodiments, use of separate optical paths 46and an optical coupler adjacent or within the housing 34 advantageouslyenables flexible placement of the red laser 134, green laser 136, andblue laser 138 so to achieve efficient implementation of the projectionsystem.

Additional Projected Content and/or Feedback Features

In many embodiments, a non-rectangular scan pattern (e.g., a spiral scanpattern, an expanding ellipse scan pattern) are used to projectavailable images having a rectangular format (e.g., video clips). Thedisplay of rectangularly-formatted content in a non-rectangularprojection area can be executed in a number of different ways, withvarious advantages. For example, the rectangular content can be croppedto match the shape of the projection area, such that the entire scan isfilled with the rectangular format content. Additionally, the croppingcan be dynamically adjusted to show different portions of the image overtime (e.g., if a circular projection is used to project a slide show ofrectangular photographs, the circular “window” can pan or scan acrossthe total rectangular image over time).

As an alternative to cropping the content to fit the shape of theprojection area, the content can be reduced in size to fit within thenon-rectangular projection area. In such a configuration, portions ofthe total projection area will be unused for the rectangular content.For example, if a rectangular image is scaled to fit within a circularprojection area, such that its corners align with the circumference ofthe circular scan, there will be four portions of the circularprojection area (round on one side and flat on the other) that are notbeing used to display the rectangular content. To make use of this freespace, as illustrated in FIG. 15A, the device software can fill thisspace with additional content, such as context-appropriate informationabout the primary content or controls for the device. For example, if arectangular movie clip 140 is being presented within a circularprojection area, the four “fringe areas” 142 can display play 144,pause, stop, fast forward 146, and rewind buttons 148 for the movie, thetitle of the movie 150, the time elapsed or time remaining in the clip152, etc. As another example, if a rectangular webpage is viewed in acircular projection area, the fringe areas can display the URL of thewebpage; back, forward, history, bookmarks, etc., buttons, and otherinformation relevant to the webpage. Additionally, these fringe areascan be used to alert the user of information that does not directlypertain to the rectangular content currently being viewed. For example,while a viewer is editing a rectangular spreadsheet in a circularprojection area, the fringe areas can be used to alert the user thathe/she has received a new email and/or to display the current time anddate. The fringe areas can also be used for user input functions thatare enabled by the generated feedback signal. For example, the fringeareas can be used for user feedback purposes, with our without thedisplay of any additional content the fringe area. For example, thefringe areas can be made sensitive to “key presses” by the user touchinghis/her finger to the fringe area or “scrolling” his/her finger across afringe area, even if “buttons” or “scroll bars” are not rendered inthose areas.

Device Orientation Compensation

In many embodiments, device orientation compensation is used todynamically orient and/or position a projected image. A projected imagecan be dynamically oriented and/or repositioned within a scan pattern inresponse to a movement of the projection device relative to theprojection surface and/or viewer. For example, when the projectiondevice is rolled relative to the projection surface, this movement canbe detected and used to reposition the projected image within theprojection area, such that the content remains horizontally aligned andstable even when the projection device is rolled and tilted (asillustrated in three sequential time steps shown in FIG. 15B). In thefirst time step, the projection device is held level, with the marker154 at the top of the circle signifying the orientation of theprojection device. In the second time step, the projection device isrotated ten degrees, and the rectangular sub-image 156 is shown as beingdisplayed at an angle (pre-compensation). The curved arrow 158illustrates the rotation of the projection device and the projectionarea. The marker 154 on the circle is to the left of center. In thethird time step, the projection device remains rotated at ten degrees,and the rectangular sub-image 156 has been rotated by negative 10degrees to compensate for the projection device rotation. The smallarrow 160 illustrates the rotation of the rectangular sub-image 156. Themarker 154 on the circle remains to the left of center because theprojection device is still rotated 10 degrees relative to the startingposition. Such minor rolls, tilts, tips, and pans can be commonespecially when the device is hand-held by a user, and it can be veryvaluable to stabilize the image. Device movements, such a device roll,can be compensated for in a variety of ways. For example, with deviceroll, compensation can be provided by a simple rotation of the imagedata in a memory buffer.

Projection device movement can be detected in a variety of ways. Forexample, the movements of the device can be detected using the feedbackfunction described above. The movements can also be detected using oneor more orientation sensors. For example, a tilt sensor or accelerometercan be used to detect a rolling of the device relative to the projectionsurface, and the projected image can be remapped within the scan patternto compensate for this rolling and keep the image properly oriented(e.g., level with the horizon). If additional content is displayed inthe “fringe areas” as described above, this content can simply berotated along with the main image, in order to keep the entire displayedimage properly oriented.

The following examples are intended to illustrate but not limit thepresent invention.

EXAMPLES

Prototype light-scanning engines that use a vibrating optical fiber toscan light in two axes were developed. Rather than reflecting light froma scanning element, the light is deflected directly as it emerges frombeing transmitted along the optical fiber, enabling a reduction in totalscanner size. Initial prototypes were used to develop wearable displaysfor users with low vision; however, a raster-scanning approach using twoorthogonal piezoelectric actuators limited display resolution to 100×39pixels. Subsequent development focused on using scanning fiber enginesto create ultra-thin endoscopes and dramatic improvements to the corefiber-scanning technology have been achieved.

A proof-of-concept compact monochrome scanned-laser projection displayprototype using an improved scanning fiber engine capable of producing500 line by 500 pixel images with a maximum throw angle of 100° wasdeveloped. The scanner is dynamically configurable; the scan angle canbe adjusted dynamically to accommodate different screen sizes atvariable projection distances, and resolution can be traded for framerate to optimize for high resolution still images or high frame ratevideo presentation.

The miniature projector vibrates the tip of a fiber optic in two axes tospirally-scan a spot of light that is luminance-modulated to form aprojected image. The scanned light is nearly collimated, enabling clearimages to be projected upon both near and distant surfaces. The totalsize of the scan engine, including lenses and outer housing, is 1.07 mm(diameter) by 13 mm (length).

At the heart of the scanning engine is a fused silica optical fiber thathas been inserted through a hollow piezoelectric actuator (PZT 5Amaterial), such that a short length of fiber protrudes from the tip ofthe actuator to form a flexible cantilever (see, e.g., FIG. 2). Thepiezoelectric actuator vibrates the fiber cantilever at its first modeof mechanical resonance. This mechanical resonance amplifies a smallvibration at the tip of the actuator by several hundred times to vibratethe tip of the optical fiber through a large scan angle (producing afinal throw angle of up to 100° after the lens system), and smaller scanangles can be produced with lower drive voltages to the piezoelectricactuator.

Quadrant electrodes, plated on the 0.4 mm diameter piezoelectric tube,enable the driving of two orthogonal axes of motion at the fiber tip. Anamplified sine wave to one axis and a cosine wave to the other axis ofthe actuator generate a circular scan. Modulating the amplitudes of thedrive signals creates an area-filling two-dimensional spiral scanpattern (see, e.g., the right side of FIG. 3).

The optical fiber has a 125 μm cladding diameter and a cantilever lengthof 4.3 mm, producing a first-mode resonance of approximately 5 kHz. Theactuator is driven at or near this resonance frequency with one axis 90degrees out of phase with the other to form a circular scan. A spiralscan of 250 rings is generated by increasing the voltage linearly to amaximum voltage defining the maximum scan angle for that frame. Thefiber is driven back to zero during the retrace portion of thetriangular shaped, amplitude-modulated sine wave, as illustrated in,e.g., FIG. 3. A short settling time is used between individual frames ofscanning at fiber resonance.

The scan angle of the scanning fiber projection display can be adjustedby varying the maximum drive voltage to the piezoelectric actuator. Forvery short throw distances, e.g., for shallow rear-projection displays,a maximum drive voltage can be used to produce a throw angle of up to100°. For front projection from a mobile device, a more modest throwangle (e.g., 30°) may be produced by using a lower drive voltage.Additionally, this scan angle may be adjusted dynamically by the deviceto create a desired image size at varying projection distances. Forinstance, a user browsing the internet at a desk placed against a wallmay use a wider scan angle to produce a large image at a relativelyshort throw distance, and a narrower scan angle to project apresentation on a distant wall at a meeting.

For this initial proof-of-concept demonstration, a luminance-modulatablediode laser (Melles Griot, model 57 ICS 072/S) was coupled to the baseof the single-mode optical fiber to project red light from the fibercantilever tip. Pixel-modulated light was projected through the fiberfor 50 ms each frame, with an additional 83 cycles per frame for retraceand settling, to project images at 15 Hz refresh rate. 2000 pixels wereprojected for each of the 250 concentric scan rings at a constant 10 MHzpixel rate, creating an equivalent resolution of 500 lines by 500pixels, though the limited modulation speed of the laser diode producedsome blurring between adjacent pixels. Sample binary red images wereprojected by the scanning fiber projection display. For these images, amoderate drive voltage (approximately +/−15 V) to the piezoelectricactuator was used in order to produce an approximate 30° throw angle.

The current prototype has demonstrated the equivalent of a 500 line by500 pixel resolution (250 concentric circles of 2000 pixels each) at a15 Hz refresh rate. To increase the refresh rate to 30 Hz whilemaintaining this resolution, the first-mode resonance of the fibercantilever must be increased from 5 kHz to 10 kHz. The resonant scanrate is a function of the material of the fiber, its radius, and thecantilever length. The relationship of the first and second resonancefrequencies (F) of transverse vibration for a cylindrical optical fibermaterial and physical dimensional properties of a base-excitedfixed-free cantilever are expressed in equation 1, below.

$\begin{matrix}{F = {\frac{\pi\sqrt{E}}{16\sqrt{\rho}}{R/{L^{2}\left( {1.194^{2},2.988^{2},\ldots}\mspace{14mu} \right)}}}} & (1)\end{matrix}$

Where ρ=density, E=modulus of elasticity, R=radius, and L=length ofsolid cylindrical fiber cantilever.

In order to achieve a first-mode resonance of 10 kHz, one may use acustom optical fiber with an 80 μm cladding diameter (StockerYale Inc.,Salem, N.H.) with a cantilever length of 2.4 mm. Using this method,scanning fiber endoscopes capable of sampling 500 line×500 pixel imagesat a 30 Hz frame rate were successfully constructed. The decreased fibercantilever length also allows a reduction in the length of the housingfrom 13 mm to 9 mm.

With a given first-mode resonance of the scanning fiber, scan lines andframe rate may be traded off with one another. The same 10 kHzfirst-mode resonance scanner has been used to generate 240-line imagesat 60 Hz and up to 600-line images at 30 Hz and, by reprogramming thedrive electronics, 1000-line images at 15 Hz can also beachieved-approximating HDTV resolution. Such configurability couldenable the user to choose a higher resolution mode when projecting stillimages and a higher frame rate when projecting video. The ultimateresolution limit is determined by the diffraction limit of the lensassembly and the type of optical scanning through the lenses, assumingcontrol of lens aberrations and scan distortions.

In conclusion, a light-scanning engine that is compact and robust wasdeveloped. A powerful feature of the scanning fiber technology is itsconfigurability. The resolution, frame rate, and throw angle can beadjusted dynamically to accommodate high resolution still images, highframe rate video presentation, and different screen sizes at variablethrow distances.

Fast modulatable laser sources in red, green, and blue colors are keycomponents for the future development of a full-color scanning fiberprojection display. Laser diodes can satisfy these requirements for redand blue (e.g., 635 nm Thor Labs Model LPS-635 and 440 nm Nichia ModelNDHB510APAEI, respectively). As there are no green laser diodescurrently available, a green semiconductor laser, such as those producedby Corning and Novalux may be used. Each laser can be coupled to asingle-mode optical fiber and a fiber-optic combiner (e.g., SIFAM FibreOptics Ltd., 635/532/460 RGB combiner) can be used to form an RGB beamwith single-mode properties.

It is understood that the examples and embodiments described herein arefor illustrative purposes and that various modifications or changes inlight thereof will be suggested to persons skilled in the art and are tobe included within the spirit and purview of this application and thescope of the appended claims. Numerous different combinations arepossible, and such combinations are considered to be part of the presentinvention.

What is claimed is:
 1. An improved method for projecting one or moreimages onto a surface with a projection device comprising alight-scanning optical fiber, the method comprising: generating asequence of light in response to one or more image representations and adrive signal determining a scan pattern of the optical fiber;articulating the optical fiber in the scan pattern based on the drivesignal, wherein a distal tip of the fiber has a variable velocity; andprojecting the generated sequence of light from the articulated opticalfiber to form the one or more images on the surface, wherein the fiberis scanned within a Q factor of a resonant frequency of the opticalfiber, wherein the generating of the sequence of light comprises varyinga light intensity based on the drive signal such that the lightintensity is varied based on both a position in the one or more imagerepresentations and a positional rate of change of the optical fiber inthe scan pattern and wherein the intensity increases based on thevariable velocity increasing.
 2. The method of claim 1, wherein the scanpattern comprises at least one of a spiral pattern, an expanding ellipsepattern, or a Lissajous pattern.
 3. The method of claim 1, wherein thescan pattern in a single frame comprises a first portion providing afirst resolution and a second portion providing a second resolution thatis different from the first resolution.
 4. The method of claim 1,wherein varying the light intensity comprises at least one of varying anamount of intensity of light in the sequence of light or varying a lightpulse duration in the sequence of light.
 5. The method of claim 1,further comprising: articulating the optical fiber in a second scanpattern in a second frame, the second scan pattern having a differentthrow angle than the first scan pattern in a first frame; generating asecond sequence of light in response to a second one or more imagerepresentations and the second scan pattern; and projecting the secondsequence of light from the articulated optical fiber to form the secondone or more images on the surface.
 6. The method of claim 1, furthercomprising: articulating the optical fiber in a second scan pattern in asecond frame, the second scan pattern providing a different resolutionthan the first scan pattern in a first frame; generating a secondsequence of light in response to a second one or more imagerepresentations and the second scan pattern; and projecting the secondsequence of light from the articulated optical fiber to form the secondone or more images on the surface.
 7. The method of claim 1, whereingenerating the sequence of light comprises buffering an imagerepresentation in a memory device and reading the buffered imagerepresentation according to the scan pattern.
 8. The method of claim 1,wherein: the images include video images; and the method furthercomprises adjusting the number of settling cycles to match the videoframe rate.
 9. The method of claim 1, further comprising: determining anorientation of the projection device; and modifying the projected lightoutput based on the determined projection device orientation.
 10. Themethod of claim 1, wherein: the projected one or more images comprise afirst portion and a second portion; the first portion corresponding tothe one or more image representations; and the second portion isprojected by a portion of the scan pattern that is not used to projectthe first portion, the second portion including additional imagecontent.
 11. A system for projecting one or more images onto a surface,the system comprising: a projection device comprising a light-scanningoptical fiber, the projection device operable to project a sequence oflight from the light-scanning fiber onto a surface according to a scanpattern; a laser assembly coupled with the light-scanning optical fiberand operable to generate the sequence of light; and a processor coupledwith the light-scanning assembly and the laser assembly, the processorcomprising a tangible medium, the tangible medium comprisinginstructions that when executed cause the processor to: generate thesequence of light in response to one or more image representations andthe scan pattern comprising a varying light intensity based on a drivesignal such that light intensity is varied based on both a position inthe one or more image representations and a positional rate of change ofthe optical fiber in the scan pattern, articulate the optical fiber inthe scan pattern based on the drive signal, in order to scan the opticalfiber within a Q factor of a resonant frequency of the optical fibersuch that a distal tip of the fiber has a variable velocity; and providecontrol signals to the laser assembly and the light-scanning assembly toincrease the intensity based on the variable velocity increasing inorder to project the sequence of light according to the scan pattern toform the one or more images on the surface.
 12. The system of claim 11,wherein the laser assembly comprises: at least one of a red laser, agreen laser, and a blue laser; and a combiner to combine output from theat least one of a red laser, a green laser, and a blue laser, whereinthe combined output is projected through the light scanning fiber. 13.The system of claim 12, wherein at least one of the red laser, the greenlaser, or the blue laser are remotely located relative to the combinerand coupled with the combiner via separate optical paths.
 14. The systemof claim 11, wherein a formed image comprises a first portion comprisinga first image resolution and a second portion comprising a second imageresolution that is different from the first resolution.
 15. The systemof claim 11, further comprising a memory buffer to store an imagerepresentation, and wherein the processor reads the buffered imagerepresentation according to the scan pattern.
 16. The system of claim11, wherein the processor generates the sequence of light by varying anamount of light in the sequence of light in response to a positionalrate of change of the optical fiber in the scan pattern.
 17. The systemof claim 11, wherein the processor can vary the scanning of thelight-scanning fiber so as to provide different throw angles.
 18. Thesystem of claim 11, wherein the processor can vary the scanning of thelight-scanning fiber so as to provide different image resolutions. 19.The system of claim 11, wherein the processor can vary the scanning ofthe light-scanning fiber so as to match a video frame rate.